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Abstract — Transformation-invariant analysis of signals often 
requires the computation of the distance from a test pattern 
to a transformation manifold. In particular, the estimation of 
the distances between a transformed query signal and several 
transformation manifolds representing different classes provides 
essential information for the classification of the signal. In 
many applications the computation of the exact distance to 
the manifold is costly, whereas an efficient practical solution 
is the approximation of the manifold distance with the aid 
of a manifold grid. In this paper, we consider a setting with 
transformation manifolds of known parameterization. We first 
present an algorithm for the selection of samples from a single 
manifold that permits to minimize the average error in the 
manifold distance estimation. Then we propose a method for the 
joint discretization of multiple manifolds that represent different 
signal classes, where we optimize the transformation-invariant 
classification accuracy yielded by the discrete manifold represen- 
tation. Experimental results show that sampling each manifold 
individually by minimizing the manifold distance estimation 
error outperforms baseline sampling solutions with respect to 
registration and classification accuracy. Performing an additional 
joint optimization on all samples improves the classification 
performance further. Moreover, given a fixed total number 
of samples to be selected from all manifolds, an asymmetric 
distribution of samples to different manifolds depending on their 
geometric structures may also increase the classification accuracy 
in comparison with the equal distribution of samples. 

Index Terms — Manifold discretization, transformation mani- 
folds, manifold distance, pattern transformations, pattern classi- 
fication 



I. Introduction 

COMPARED to traditional signal processing techniques 
where the treatment of signals is performed in the high- 
dimensional signal space, the characterization of signal sets 
via low-dimensional manifold models have several advantages. 
Particularly in image processing, manifold models have been 
seen to provide more compact and efficient signal representa- 
tions as well as assisting the analysis of data. Manifold models 
for signals have recently been studied in several research areas 
such as dimensionality reduction HI, |[2|, image registration 
0, (H and transformation-invariant pattern classification (31 . 

A signal manifold in a high-dimensional signal space is 
a set of signals that can be mapped to a lower-dimensional 
parameter space. In |6| Peyre et al. demonstrate several ex- 
amples of parametrizable signals using local patch manifolds, 
and examine inverse problem regularizations with manifold 
models and applications to image inpainting. Similarly, Wakin 
et al. study image appearance manifolds (lAM) in |7|, which 
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are image sets defined by a few generating articulation pa- 
rameters. In this work, we consider transformation manifolds 
that are defined as signal sets which can be described by 
parametrizable transformations of a reference signal model. 
An instance of transformation manifold can be a pattern 
transformation manifold, which refers to the family of patterns 
obtained by applying a certain set of geometric transformations 
to a reference pattern; or the observation manifold of an 
object, which consists of its images captured under varying 
viewpoints. Among several manifold models, transformation 
manifolds are of specific interest as they provide a basis 
for performing transformation-invariant analysis of signals 
that have been exposed to geometric transformations. For 
instance, it is possible to classify a transformed query signal 
by computing the distance between the signal and each of the 
transformation manifolds that represent different classes. The 
class label of the signal is then estimated to be the same as 
the class label of the manifold that has the smallest distance 
to the signal. 

The computation of the manifold distance is in general a 
demanding task, mainly due to the variety and complexity 
of the involved models. There have been several research 
efforts addressing the manifold distance computation problem. 
The tangent distance provides an estimation of the manifold 
distance through local linearity assumptions |8|. However, 
this approach is likely to suffer from local minima, which 
is improved by the investigation of the tangent distance in a 
multiresolution manner fS]. While these are generic methods 
applicable to a large set of manifold-modeled signals, there 
are also some works proposing transformation estimation 
solutions for particular types of manifolds, such as those 
generated by 2D pattern transformations 141. l|9l. 

Although there have been many attempts, the general prob- 
lem of manifold distance estimation still retains its challenges 
as it usually involves high computational complexity. In order 
to estimate the manifold distance, it is a practical solution to 
represent the manifold by a finite grid of manifold samples, 
where the distance between a space point and its projection 
onto the manifold is approximated by the distance between the 
point and the nearest manifold sample. The usage of such a 
grid improves the distance estimation complexity immensely, 
possibly at the price of a lower distance accuracy. 

The choice of the manifold grid has considerable influence 
on the accuracy of the distance estimation. In this work, we 
study the distance-based discretization of signal manifolds 
of known parameterization. We build on our previous work, 
where we propose a manifold discretization algorithm that 
minimizes the manifold distance estimation error stemming 
from the representation of the manifold by finitely many grid 
points 1 10 |. Our discretization method bears some resemblance 
to the LBG vector quantization algorithm ifTTl due to the al- 



2 



ternating steps of optimization that compute the representative 
samples for a given space partition, and respectively the parti- 
tion for a given set of samples. However, the proposed method 
differs essentially from the LBG algorithm, since it targets the 
minimization of the manifold distance with samples positioned 
on the manifold and does not have a signal approximation 
objective. Noting the dependency between the registration and 
classification efficiencies, we extend this sampling solution to 
a setting with several transformation manifolds representing 
different class models, where we optimize all manifold sam- 
ples in a joint manner such that the relative geometries of 
different manifolds are taken into account. The experimental 
results show that our discretization approach is significantly 
better than basic discretizations such as random grids or 
regular grids in the parameter domain in terms of registration 
and classification accuracy. The classification performance is 
further improved when the relative properties of the manifolds 
are also considered in addition to their individual properties 
in the sampling. 

Meanwhile, some state-of-the-art solutions to surface dis- 
cretization are as follows. It is typical to achieve the sampling 
in a straightforward way by generating a grid regular in the 
parameter domain. However, especially when the number of 
manifold samples is limited, a regular discretization in the 
parameter space is not guaranteed to offer a good performance. 
Structured grid generation has been well-studied especially 
for analytical two-dimensional surfaces in R^, mostly for 
the purpose of obtaining finite-difference solutions to partial 
differential equations |12|. On the other hand, it is also 
possible to find sampling solutions for surfaces represented in 
non-analytical forms such as meshes |13|. Even though some 
of these sampling methods may in principle be generalized 
for arbitrary dimensional manifolds, the targeted applications 
must be taken into account in grid generation. 

The paper is organized as follows: In Section |llj we for- 
mulate the utilization of transformation manifolds in signal 
classification and set our notation. In Section [III] we overview 
the discretization of parametric signal manifolds based on 
distance estimation, whereas in Section [IV] we propose an 
extension of the registration-based sampling solution for clas- 
sification. We present the experimental results obtained on 
some transformation manifolds in Section |V| and conclude 
in Section 

II. Manifold Models in Transformation-Invariant 
Signal Analysis 

The analysis of transformed signals in registration or classi- 
fication settings mostly consists in estimating their distance or 
projection to transformation manifolds. Let M C denote 
a set of signals conforming to a manifold model defined 
over the parameter domain A C M^. Then, M is given by 
M = {/7(A), A G A}, where U denotes the mapping from the 
(i-dimensional parameter domain A to the set of n-dimensional 
signals. As an example, consider a reference visual pattern, 
and the family of patterns generated by applying arbitrary ro- 
tations and scale changes to the reference pattern. Then, every 
combination of rotation ijj and scale s defines a parameter 



vector A = (V^, s), and the set of constructed patterns lies on 
a pattern transformation manifold M. = {[/(A), A = 
of intrinsic dimension d = 2. Image appearance manifolds 
|7 1 or image parameter articulation manifolds lfT4ll constitute 
other examples of the manifolds we study. Although we focus 
on transformation manifolds in this work, we maintain a 
generic formulation that it is applicable to parametric signal 
manifolds in general. We also note that parametrizable signal 
manifolds are not restricted to image manifolds, which could 
find examples within acoustic and seismic signals for instance 

161, ca. 

Let X G be a signal belonging to the space of n- 
dimensional digital signals. We denote the distance of x to 
the manifold M as D{x,M.) = mmxeA{d{x, U{X))}, where 
the distance function d{x, y) is taken as the ^2-distance — ?/|| 
throughout this work. Then, a projection y of x onto the mani- 
fold is a manifold point with minimum distance to x, i.e., y = 
[/(A*), for some A* such that A* — arg min;\^A{<^(^7 ^('^))}- 
The projection of a signal onto a transformation manifold is 
typically associated with the registration of the signal with 
respect to a transformation model. Given a signal x with an 
unknown geometric transformation and the related transfor- 
mation manifold M, the estimation of the transformation that 
best represents x corresponds to finding the point of M with 
smallest distance to x. 

Now let us consider a setting with multiple signal manifolds, 
each of which represents the transformation model of a dif- 
ferent class. Let Mi,M2, • • • , Mm C be smootlj^man- 
ifolds of M different signal classes defined on the parameter 
domains Ai, A2, • • • , Am by the mappings /7i, U2, - • • , Um, 
where Um - A^m, m = 1, • • • , M. We assume that all 

{Mm} are submanifolds of the same Euclidean space W^, 
however, the parameter domains {A^} may be subsets of 
different spaces. Let x G M"^ be a query signal of unknown 
class that has undergone an unknown transformation. Then, 
provided that the transformation manifolds {Mm} constitute 
sufficiently accurate models for the representation of the 
involved signal classes, x belongs to the same class as the 
manifold with minimum distance to it, i.e., the class label l{x) 
of X is given by 

l{x) — argminl)(x, Mm)^ m = 1, • • • , M. (1) 

m 

Now let us consider only two manifolds Mm and Mr 
representing two different class models. In order to represent 
the sets of signals that belong to the same class as each of 
these two manifolds, we define the half-space^ 1-Lmr and 
Hrra as follows! We denote by Hmr the set of points whose 
distance to is smaller than their distance to A^^ (notice 
that Hmr 7^ Hrm with our notation): 

Definition 1: 

Hmr = {X G : D{x,Mm) < D{x,Mr)}- (2) 

^In (7] it is shown that differentiability of a manifold fails in the cases 
where the generating model involves sharp edges, however, approximate 
differentiable representations of such manifolds can be obtained by their 
regularization. 

^Although in the standard definition, the boundary surface determining a 
half-space is an affine hyperplane, here we generalize the term to include the 
case where the boundary is a hypersurface. 
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The notation in ([2|) denotes the closure of the set S. Note 
that Hmr is defined in this way considering the degenerate 
cases that may be caused by manifold intersections. We then 
define the decision surface Bmr as the boundary of the half- 
space l~Lmr-> 
Definition 2: 

Smr — dl~Lrnr- (3) 

Bmr is a combination of hypersurfaces, i.e. a union of (n — 1)- 
dimensional manifolds in R^. 

If we consider M class representative manifolds instead of 
two, the set of space signals belonging to the class represented 
hy Mm are given by the approximation region Hm C of 

Mm' 

Definition 3: 

= ^mr • (4) 

rG{l,--- ,M}\{m} 

Finally, in order to adapt the decision surface Bmr determined 
by two manifolds to the case of multiple manifolds, we define 
the combined decision surface B, 
Definition 4: 

M 

where the notation dS denotes the boundary of the set S. 
The combined decision surface S is a collection of subsets 
of all decision surfaces Bmr, i-e., B C Umy^r^rnr- It forms 
a boundary between different portions of the space, where 
each of these portions belongs to a different class as imposed 
by the manifolds Mi^ M2r ' ' ,A^M-An illustration of class 
representative transformation manifolds, the approximation 
regions Hm of manifolds, and the decision surface B is given 
in Fig. [T] 

Now, having set our notation, we remark the following. 
The registration problem consists in obtaining the projection 
of a signal x onto a manifold M, and the classification of 
the signal x corresponds to the determination of the approx- 
imation region Hm it lies in. The exact knowledge of the 
manifolds determines the projection and class label of a signal 
perfectly. Although a discrete representation is less accurate, 
it reduces significantly the complexities of the registration and 
classification problems, and the requirement for storage space. 
The accuracy of the discretization depends on its capability in 
approximating the manifold distance. In Sections |lll| and |lVj 
the registration and classification accuracies of discretizations 
are formulated and sampling solutions are proposed. 

III. Manifold Discretization for Minimal 
Distance Estimation Error 

In this section, we describe a solution for manifold dis- 
cretization such that the manifold distance estimation error 
caused by representing the manifold with a finite number 
of samples is minimized. Note that the main purpose of the 
sampling scheme proposed here is the accurate estimation of 
the manifold distance for registration applications rather than 
the approximation of the manifold, which are not necessarily 




Fig. 1. Illustration of transformation-invariant signal classification via 
transformation manifolds: A4i, are three manifolds representing 

different classes; 1-Li, H2, and H3 are their respective approximation regions; 
and B is the combined decision surface. 

equivalent. We build on fTOl and present an iterative approach 
for the optimization of the manifold samples. 

The discretization of a manifold M consists in selecting a 
predetermined number N of manifold points, i.e., a sample 
set S = {Si} = {U{Xi)} C M, i = I,-- - ,7V for some 
{Ai,--- ,Xn} C a. 

We would like to select a set of samples that minimizes the 
total manifold distance estimation error E over R, where R is 
a bounded and piecewise connected region in the space R^. 
We consider to be a region of interest which depends on the 
application. For instance, in the case of images one can define 

as a hyperrectangular region R= {x eW^ : < Xi < bi} 
with the typical values of and 255 for the parameters and 
bi. We define the error E by 

E = [ {D^{x,S)-D^{x,M))dx, (6) 
Jr 

where D{x^S) = min.i^i^2,--- ,N{d{x^ Si)} denotes the dis- 
tance between x and the sample set S. The formulation of the 
error in terms of the squared distances is due to the ease of 
analytical manipulation. 

For a given sample set, one can partition R into N regions 
as = Ui^i where each Ri is a region consisting of 
points with smallest ^2 -distance to Si among all samples, i.e., 
Ri = {x e R : d{x,Si) < d{x,Sj),\fj G {!,••• ,N}}. 
Hence, the total manifold distance estimation error becomes 

N ^ r 

E = ^Ei = ^ {d^{x,Si)-D^{x,M))dx. (7) 
i=i i=i -^^i 

In order to minimize the error E, we follow an iterative 
optimization procedure. In each iteration of the algorithm, 
we first determine the partition regions corresponding to the 
samples, and then optimize each sample individually such 
that the error Ei in the regarding region is minimized. Once 
the partition is determined, the minimization of the manifold 
distance estimation error Ei within a specific region Ri is 
achieved as follows. The error term Ei can be rearranged as 

Ei= d^{x,Si)dx- / D'^{x,M)dx, 

JRi JRi 

where the second integration depends only on Ri, and is 
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constant with respect to Si. Therefore, Ei is given by 

Ei = (f {x^ Si) dx -\- Ci 

JR^ 

= / x^xdx-2Sj / xdx^ViSfSi^Ci, 

JRi JRi 

where Ci is a constant independent of Si, and Vi = dx is 
the volume of the region Ri. Denoting the centroid of Ri by 

Gi = {j^^xdx)/{j^^ dx), we get 

Ei = I x^xdx^V^{-2SjG,^SjS,)^c, 
= V^{SjS^-2SjGi)^c',, 

where we express the sum of the terms independent of Si 
by As Ei differs from \\Si — Gi\\^ only up to a positive 
multiplicative factor and an additive term constant with respect 
to Si, one can equivalently minimize 

e^ = \\S,-G£ (8) 

at each iteration of the algorithm. This actually means that Si 
should be selected as the manifold point closest to the centroid 
of the region Ri. 

The following is a summary of the procedure we apply for 
obtaining a manifold discretization that minimizes the total 
manifold distance estimation error. Given the available domain 
of parameters and the mapping defining the manifold, we begin 
with an initial sample set 5(0) = {6'i(0)} on the manifold, 
which is possibly randomly selected. We optimize the sample 
set iteratively. In each k^^ iteration of the algorithm, we first 
compute the regions {Ri{k)} that partition R with respect 
to the manifold samples {Si{k)}, and then we modify each 
sample Si{k) individually to obtain the new sample Si{k^l) 
such that the manifold distance estimation error given by ([8| 
is minimized in the corresponding region. The new sample 
+ 1) is the projection of the centroid Gi{k) onto the 
manifold. Iterations are repeated until improvements become 
negligible. We call this algorithm Registration-Efficient Man- 
ifold Discretization (REMD). An iteration of the algorithm is 
illustrated in Fig. [2] and the pseudocode is given in Algorithm 

m 

Assuming that the feasible domain A of parameter vectors is 
compact and the mapping U is bounded, for a given number of 
samples N, there exists a solution <S* that globally minimizes 
the total error E m Ai each iteration of the discussed 
method, first the partition regions are updated and then the 
samples are readjusted, both of which are modifications that 
either reduce E or retain it. Since the error E is non-increasing 
throughout the iterations and is also lower bounded, the 
algorithm converges. However, in general the cost function is 
a non-convex, complicated function of the parameter vectors; 
therefore, the algorithm is not guaranteed to converge to 
the globally optimal solution. This could be mitigated by 
the choice of a good initial distribution of samples. For 
instance, in order to begin with a fair and balanced sample 
distribution, a preliminary stage can be added before the main 
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Fig. 2. Illustration of a single iteration of the algorithm with three samples: 
Each Gi(k) is the centroid of the partition region corresponding to the sample 
Si(k) at the k^^ iteration. The updated sample Si(k + 1) is the projection 
of the centroid Gi{k) onto the manifold. 



iterations. Here one can impose the condition that the pairwise 
distance between any two samples in the ambient space or the 
parameter space is larger than some threshold value. 



Algorithm 1 Registration-Efficient Manifold Discretization 



IV. Classification-Based Discretization of 
Multiple Manifolds 

We have examined above a discretization solution for a sin- 
gle signal manifold based on the minimization of the distance 
estimation error given by the approximation of the manifold 
by a set of samples. Now we consider the sampling problem 
with multiple signal manifolds. We consider that the manifold 
distance is computed with a discrete set of samples from each 
manifold, and the estimated class label of a signal is the label 
of the nearest manifold sample. Clearly, the accuracy of a 
sampling in classification is highly related to its accuracy in 
distance approximation. One possible solution to the multiple 
discretization problem is to sample each class representative 
manifold independently with the REMD algorithm reviewed 



1: Input: 

A: Feasible domain of parameter vectors 
U : Mapping from parameter domain A to manifold M 
N: Number of manifold samples 
2: Initialization: 

3: Choose an initial set of manifold samples 5(0) = {5'i(0)}, 
z = l,--- ,7V. 

4: k = 0. 

5: repeat 

6: Determine the partition regions {Ri{k)}. 

7: Compute the centroids {Gi{k)} of the regions. 

8: Update each sample Si{k) to Si{k + 1), which is the 

projection of the centroid Gi{k) on the manifold. 
9: k = k^l. 

10: until The difference between S{k) and S{k — 1) is 

insignificant 
11: S = S{k). 
12: Output: 

5 = {S'i}: A set of manifold samples 
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in Section III Although this brings a certain improvement in 
the classification performance compared to baseline sampling 
solutions such as regular or random sampling, it fails in taking 
into account the geometric properties of different manifolds 
relative to each other. Therefore, a preferable approach to the 
multiple discretization problem relies in the joint discretization 
of all manifolds. 

Furthermore, given a fixed budget for the total number of 
manifold samples, which can also be interpreted as a fixed 
computational complexity for classification, we would like 
to determine how many samples should be selected from 
each manifold such that the overall classification accuracy is 
maximized. The convenience of representing a manifold by a 
discrete sample set is highly dependent on the manifold geom- 
etry. The sample budget may thus vary for different manifolds. 
Moreover, in the determination of the budget allocation, the 
relative dependencies of the manifolds must also be taken into 
account. For instance, if a subgroup of manifolds are more 
likely to lead to misclassifications because of their internal 
resemblance, then it may be more preferable to allocate them 
a higher number of samples. 



In Section |IV-A we first analyze the classification accuracy 



with a multiple manifold discretization, then in Section IV-B 
we describe an iterative algorithm for sampling multiple signal 
manifolds that aims to improve the classification accuracy 



gradually. In Section IV-C we discuss some possible ap- 
proaches for determining the allocation of the overall sample 
budget to different manifolds. 

A. Classification with Discrete Samples on Manifolds 

The classification of signals with discretized manifolds can 
be formulated as follows. We consider M signal manifolds 
Ml, ' ' 5^M, where each manifold is approxi- 
mated by the finite set 

5™ = {Sn = {U^iXT)} c M„, (9) 

of Nm samples, for i = 1, • • • , Nm and G A^. The classi- 
fication problem consists in determining the closest manifold 
for each test signal. Given a signal x e W^, the estimation 
l{x) of its true class label l{x) is determined by its nearest 
neighbour among all manifold samples: 



l{x) = aig mm D{x,S^) 



arg mm 



min d(x,Sl^) 



(10) 



We analyze now the classification error induced by the 
discretization of manifolds. Let R C denote a bounded 
and piecewise connected region of interest in the signal space. 
For each manifold Atrn, we define a partitioning of R into 
regions {Rf^}, where each region consists of points closest to 
a specific sample S^^ of among its all samples, 

(11) 

R"^ = {xeR: d{x, SD < d{x, Sf), 



Now, consider a signal x G R^^ that is of the class given by 
Mm^ i-e., X e Rf^ n l-Lm- Depending on the distribution of 
samples, x can be correctly classified only if its distance to 
is the smallest among the distances to all other manifold 
samples. Thus, we define a function : R^ {0? 1} such 
that it represents the classification error for signals of class m 
in the region R^: 



{ 



1 ifxGH^and d{x,S^)>D{x,[j^^^S-) 



otherwise 



(12) 



Then, from ^V2\ we define the total classification error E, 



M N, 



I -Dm 



(13) 



Notice that due to the definition of the error E^{x), the total 
classification error E corresponds to the sum of the volumes 
of the regions in R where signals are not correctly classified. 
Another source of misclassification associated with a sample 
corresponds to the points that are actually closer to another 
manifold than Mm^ but are misclassified as a result of being 
closer to S]^ than their nearest manifold sample of the correct 
class. Hence, in analogy with E^, we can define an alternative 
classification error function F-^ : R^ {^A} as 



Frix) 



1 ifxiU^ and d{x, Sr) < D{x, S^) 



otherwise 



(14) 

which leads to the following alternative formulation of the 
total classification error F\ 



M Nrr 



(15) 



The classification errors E in ([13]) and F in ( p3] ) are equal. 
However, due to the two mentioned sources of misclassifica- 
tion associated with a single sample S^^, we formulate the 
classification error as the combination of the two, where the 
reason for this choice is made more clear in the algorithm 



description in Section IV-B Hence we write the total classifi- 
cation error as 

^ = 9 E E / (^r(a^) + Frix)) dx, m 

m=l i=l -^^i 

where s = E = F. 

Note that a geometric interpretation of the problem is the 
following. Let R^ denote the region of space with smallest 
distance to the sample S^^ among the samples of all manifolds: 



= {x G : d{x, SD < d{x, ^J), 

VrG{l,--- ,M},Vj G{1,--- ,Arj}, 



(17) 



where R^ C R^. Also, let A^j^ denote the affine hyperplane 
which is at equal distance to the two samples S^^ and Sj 
from two different manifolds Mm and A^^. In analogy with 
the way that the decision surface B is defined in Section 
[ll| as the boundary for the determination of the true classes 
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Fig. 3. Illustration of transformation-invariant signal classification via 
transformation manifolds 



l{x) of signals, let now A denote the decision surface which 
determines the class label estimation l{x) of signals based 
on the manifold approximations given by samples. Hence, we 
define 



M 



Nrr 



(18) 



where A C A^j^. The definition of the classification 



error function s in ([16]) corresponds to the total volume of the 
regions between the true boundary B and its approximation A. 
Therefore, the problem of minimizing the classification error 
can be regarded geometrically as the selection of the sample 
sets Um=i ^^^h Ih^l resulting A constitutes an accurate 
approximation of B inside R. This is illustrated in Fig. [3] 

B. Discretization Algorithm 

We would like to minimize the classification error e in ([16]) 
by optimizing the sample sets Um=i order to achieve 

this, we suggest an iterative procedure as follows. We start 
with an initial set of samples. Then, in each iteration we 
optimize one manifold sample S^^ and try to reduce e by 
perturbing S^^. However, the dependence of the classification 
error on the location of a sample S^^ is fairly intricate, and it 
is not simple to determine the optimal sample location. Hence, 
in the minimization of the error, we adopt a constructive 
approach rather than optimal, therefore, the search directions 
in the perturbation of a sample may not always decrease the 
overall error. In order to handle this, we accept an update on 
a sample location only if it reduces the classification error. 
After reaching a locally optimum error with the perturbation 
of the single sample S^^, we repeat this process with different 
manifold samples until convergence. The reduction of the 
error at each step assures the convergence of the algorithm. 
The overall procedure is not guaranteed to converge to the 
globally optimal solution and results in a local minimum, 
whose accuracy depends on the initialization of the samples. 

In a single iteration of the algorithm, we would like to find 
an update on that reduces the error e, where the rest of 
the samples are considered to be fixed. The examination of 



twofold. The terms EY^{x) and F^{x) involve the distance 
of space points d{x^ S]^) to the sample S^^, but the region 
of integration R^^ is also defined by the position of the 
sample S]^. Hence, the classification error has a complicated 
dependence on the sample location S^^. Let 6^ and 
denote the regions of R^^ where E'^{x) = 1 and F^{x) = 1 
respectively (see Fig. [3]): 



{xeRT\n„,: 

d{x,ST)<D{x, [j S"-)}. 



(19) 



(20) 



The error terms in the expression ([16]) contributing to e are 
in fact the sum of the volumes of these two regions 6^ and 
Therefore, in order to reduce the error we seek an 
update on S^^ that decreases the volumes of 6^ and <l>^. Let 
SY^ik) be the location of the sample S^^ at the k^^ iteration 
of the search algorithm, and let Rf{k), Qf{k), and ^f{k) 
be defined similarly. The definitions ([19]) and ( [2Q| suggest 
that decreasing the distance d{x^S^{k)) between the sample 
and the points in Q'^{k) reduces the misclassified portion 
of Q^{k). Similarly, it is necessary to increase the distance 
d{x^SY^{k)) between the sample and the points in ^Y^ik) in 
order to reduce the misclassified portion of <l>^(/c). Hence, we 
define the distance measures DQ{S^{k)) and D^{S'^{k)) as 
follows: 



er(/c) 



d^[x,S^{k))dx, (21) 



d^{x,S'^{k))dx. (22) 



Eq. (16) reveals that the effect of the sample S]^ on e is 



As discussed in Section [Hlj the minimization of DQ{SY^{k)) is 
possible by minimizing the distance d{0^ (k) ^ S^^ {k)) , where 
0'^{k) is the centroid of 9-^(/c). Similarly, in order to max- 
imize D^{S^{k)), one should maximize d{(l)Y' {k) , S^" {k)) , 
where (j)^{k) is the centroid of ^^{k). However, even an 
update on SY^{k) that decreases DQ{SY^{k)) and simultane- 
ously increases D^{SY^{k)) does not guarantee that the total 
classification error e decreases. This is because in general 
GY'ik + 1) (Z: OY'ik) and ^Y^i^ + 1) ^Ti^)- Even if 
the error is reduced within 6^(/c), 6^(/c + 1) might contain 
points that are not inside Q'^{k) and actually increase e. Still, 
when it is aimed to reduce e by perturbing only SY^{k) in a 
given configuration of the samples and manifolds, curing the 
immediate regions of misclassification 6-^(A:) and ^Y^{k) is 
a promising attempt. We thus propose to update the sample 
S^{k) in the following way as long as the overall error does 
not increase. 

Let lJi'^{k) and h'^{k) denote the parameter vectors corre- 
sponding respectively to the projections of the centroids OY^{k) 
and (l)^{k) on the manifold. Then, the purpose of moving 
S^{k) closer to 0^{k) and away from (i)'^{k) leads to the 
following two updates: 
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Srik + 1) = Um{{l - a)\f{k) +aiif{k)) 



such that a minimizes £, 



STik + 1) = Um{{i + P)\T{k) - p vr{k)) 

such that 13 minimizes 



(23) 



(24) 



where A^(A:) is the parameter vector defining S^{k) and 
both a and /3 are positive scalars. Hence we determine 
the directions of perturbation with respect to the centroids 
of the misclassified volumes, and we adjust the amount of 
perturbation to obtain the largest decrease in the error. In this 
way, we find a locally optimum sample location reducing the 
misclassified portions of 6^(A:) and ^^(/c). It also guarantees 
that the possible penalty of creating new misclassified regions 
by moving the sample is always smaller than the benefit of 
correcting previous misclassifications. In the optimization of 
a single sample S^^, we alternate between the updates in 



(23) and (24) until convergence, where the parameters lJi^{k) 
and h'i^{k) are updated after each perturbation. Then we 
continue the optimization process by picking other manifold 
samples and applying the same procedure until the classifi- 
cation error converges. We call this algorithm Classification- 
Driven Manifold Discretization (CMD) and give an overview 
of it in Algorithm [2] Finally, we note that at each iteration 
the perturbation of a manifold sample in the described way 
corresponds to a one-dimensional search in the d-dimensional 
parameter space. Although the algorithm is not guaranteed 
to be optimal, it offers a compromise in the performance- 
complexity trade-off. 

C. Sample Budget Allocation 

We have considered so far that the number of samples per 
manifold is predetermined. We address now the problem of 
the allocation of samples from a total budget to the different 
manifolds. This allocation is driven by the properties of the 
different manifolds. We propose two solutions for budget 
allocation that can be paired with the CMD algorithm. 

A first simple way of determining the sample budget be- 
tween manifolds is the following. We initialize the sample set 
of each manifold to be a dense grid on the manifold, and delete 
samples progressively until the number of total samples meets 
the budget constraint. We determine the sample to be deleted 
based on the classification error associated with each sample. 
Following the previously mentioned arguments, we delete a 
grid point on one manifold where is relatively small. 
When the number of remaining grid points reaches the budget, 
we optimize the resulting sample sets using CMD for the 
final adjustment of sample locations. Since the optimization of 
sample locations takes places after the budget allocation, we 
name this approach Manifold Discretization with Predefined 
Allocation (MDPA). 

Secondly, we introduce a joint budget allocation and CMD 
sample optimization solution where we allow the deletion 
of a sample from one manifold to create a new sample in 
another manifold during the iterations. In order to elaborate 
on the transfer of samples between manifolds, we turn back 



Algorithm 2 Classification-Driven Manifold Discretization 
1: Input: 

A^: Feasible domains of parameter vectors 

Um ' Mappings from parameter domains A^ to manifolds 

Mm, m = I,--- ,M 

[jm=i ^^(0) = [Jm=i UtiiSrm'- Initial set of man- 
ifold samples 
2: Initialization: 



3: Initialize total classification error e as defined in Eq. (16). 



4: k = 0. 

5: repeat 

6: Pick (possibly randomly) a manifold Mm and a sam- 
ple SY" from this manifold, m G {1,---,M}, i G 

7: repeat 

8: Determine the misclassified region 6^(/c) and the 

parameter vector /J^Y^{k). 
9: Update SY'ik ^l) = Um ((1 - c^) Ar(^) + a iif{k)) 

such that a minimizes e. 
10: k = k^l. 

11: Determine the misclassified region ^^(/c) and the 

parameter vector i''^{k). 
12: Update 5r(^ + l) = Um{{l^ PW{k) - p ur{k)) 

such that P minimizes e. 
13: k = k^l. 

14: until The location of sample S^^ converges 
15: until £ converges 

17: Output: 

|Jm=i ^ representative samples for each man- 
ifold 



to the geometric interpretation of our problem formulation. 
In a configuration with multiple manifolds, our definition of 
the classification error is the total volume of the regions lying 
between the piecewisely planar boundary surface A formed 
by the sample sets and the true boundary surface B. As a 
result, the classification accuracy of such a setting is directly 
dependent on the approximability of the boundary surface 
S by a piecewise linear model. A "well-behaved" manifold 
region corresponding to a well-approximable part of B is more 
amenable to be represented by a small set of discrete samples 
than a manifold region corresponding to a part of B that is 
difficult to approximate by a linear model. Therefore, the 
compensation of the loss of a sample from a well-behaved 
manifold region is relatively easier. We thus propose the 
following method for dynamic sample budget allocation and 
optimization. We start with an equal distribution of samples 
per manifold that satisfies the overall budget constraint, and 
begin optimizing them with the CMD algorithm. During 
iterations, we deduce that the manifold region around a sample 
has poor representability, whenever the misclassified region 
has a large volume compared to average and the iterations 
7p4| in Algorithm [2] fail to improve the classification accuracy. 
In a similar way, we determine regions of good representability 
around samples where the volume of the corresponding 
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is relatively small. Consequently, we add a new sample to the 
poorly representable region, at the cost of deleting a sample 
from a well representable region selected among all manifolds. 
We place the new sample at the point corresponding to the 
projection of the centroid of onto the manifold. We further 
adapt the discretization to the new configuration in the regions 
where sample deletion and creation have taken place, and we 



apply the iterations [7p4] of CMD to all neighboring samples 
on the same manifolds as the deleted sample and the newly 
created sample. Note however that the sample transfer is not 
guaranteed to reduce the classification error. In order to ensure 
the improvement of the classification accuracy, we finally 
accept the update only if it reduces the error. We call this 
approach Dynamic Manifold Discretization (DMD). 



V. Experimental Results 



A. Setup 



We now present experimental results demonstrating the per- 
formances of the manifold discretization algorithms discussed 



in Sections III and IV All experiments are conducted on two 
different kinds of image appearance manifolds, namely the 
transformation manifold of a 2D pattern, and the manifold 
generated by the observations of a synthetical 3D object under 
varying viewpoint. 

In the first experimentation setup, we construct pattern 
transformation manifolds generated by the 2D rotation and 
translation of visual patterns. Given a pattern p, we define its 
transformation manifold A4 by 



M = {U{X)p:X = {i^,U,ty)eA}, 



(25) 



where i/j is the rotation parameter, tx and ty are the horizontal 
and vertical translation parameters, and A is the domain of 
transformation parameter vectors. For experimentation, we use 
a database of top- view images of 5 different objects, where 
each object has 500 different images captured under different 
orientations and positions. An example image for each object 
is displayed in Fig. |4] Note that due to the positioning of the 
camera and the limitations on object positions, the 2D pattern 
transformation model in ( [25] ) constitutes an approximate model 
for the observations. For each of the objects we build the trans- 
formation manifold of a fixed representative pattern picked 
among the database images. The image set of each object 
is grouped randomly into 300 training and 200 test images. 
The categorization of the database into training and test sets 
is changed randomly at each repetition of the experiment. 
For computational convenience, all images are converted to 
grey scale, downsampled to a resolution of 50x60 pixels, and 
background pixels are set to the luminance value of by 
simple thresholding. Manifold points are generated by rotating 
and translating the representative pattern (cropped previously 
near the boundary) over a 50x60 pixel zero background 
within the parameter range i/j G [— 7r,7r); tx G [—7,7]; 
ty G [—12, 12]. All images and generated manifold points are 
normalized to have unit norm. Some illustrative images from 
the transformation manifold of one of the objects are given in 
Fig.|5] 




Fig. 4. Example images from database 




(a) (b) (c) (d) 

Fig. 5. Images from a pattern transformation manifold 



In the second experimentation setup, the object observation 
manifolds are generated as follows: Given a synthetical 3D 
object model P, we consider the observation manifold defined 
by 



M = {U{X)P : X = i^x.^y.^z) e A}, 



(26) 



where U{X)P is the image of the object rendered under the 
viewpoint specified by the three rotation angles ipx^i^yji^z- We 
use the Princeton Shape Benchmark database of 3D models 
fT6l, where we conduct our experiments on 8 different classes 
of objects (car, airplane, ship, tank, human, animal, table, 
bottle) with several (4-30) objects belonging to each class. 
Some example objects belonging to the airplane class are 
shown in Fig [6] For each class we choose a representative 
object, and generate the observation manifold of the represen- 
tative object in the parameter range ^l^x^i^y^i^z ^ [— 7r/4, 7r/4]. 
The representative object of each class is changed randomly 
in repetitions of the experiment. All rendered images are 
converted to greyscale, downsampled to the resolution of 
50x50 pixels and normalized to unity. The training and test 
sets for each manifold consist of 500 random observations 
of the objects of the same class within the same parameter 
range. Some images from an object observation manifold are 
displayed in Fig. [7] 

In both experimental setups, we use training images only 
for the computation of the centroids of space regions. We 
compute the centroid of a region of M.^ experimentally by 
taking random training images, checking if they are in the 
inquired region, and then computing the arithmetic average 
of inliers when a sufficient number of them are accumulated 
as suggested in il7i After the centroids are computed, we 
estimate their projections onto the manifold with the aid of 
a dense grid on the manifold. We first locate the projection 
coarsely by finding the grid point that has the smallest distance 
to the centroid, and then refine the location of the projection by 
minimizing its distance to the centroid using gradient descent 
tools. 
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Fig. 6. Example objects from the airplane class 




(a) (b) (c) (d) 

Fig. 7. Images from an object observation manifold 
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(b) Distribution of test points with respect to projection accuracy 



Fig. 8. Sampling results obtained on pattern transformation manifolds 
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Fig. 9. Sampling results obtained on object observation manifolds 



B. Results on registration accuracy 

Here we test the REMD algorithm on pattern transformation 
and object observation manifolds. In both experimental setups, 
we initialize the algorithm with a randomly selected sample 
set. We compare the sample set determined by the REMD 
algorithm to the initial random sample set and to the sample set 
given by a regular grid over the parameter domain. The perfor- 
mance evaluation criterion is the accuracy of the discretization 
in manifold distance estimation. For each discretization, the 
distances of test points to the sample set are computed and 
the average registration error is calculated. The registration 
error is taken as the ^2 -distance between the exact projection 
of the test point onto the manifold and the manifold sample 
with smallest ^2 -distance to the test point. 

In the setup with pattern transformation manifolds, we 
build and sample the transformation manifold of each object 
individually. For each object the experiment is repeated 10 
times with different random initializations. The results are 
averaged over all realizations and all objects. In Fig. |8(a)| 
average registration errors obtained with the REMD output, 
random and regular sample sets are plotted for several numbers 



10 



of samples. Then, we compute the distance of each test point 
to all manifold samples obtained with these three discretization 
approaches. For each test point we determine the sample 
among all three sets that has the smallest distance to the test 



domain with respect to their classification performances. The 
experimental setting is the same as that of Section fV-B] i.e., the 



point. In Fig. |8(b)| we report the percentage of the test points 
that have their closest manifold sample within the REMD 
output, random and regular sample sets, respectively. 

These experiments intend to measure the capability of the 
discretization to provide an accurate approximation of the 
projection onto the manifold. As shown in the figures, the 
discretization obtained by the REMD method yields the least 
registration error when compared to the random discretization 
and the regular discretization in the parameter domain. In 
addition, for the majority of the test points the most accu- 
rate approximation of the projection lies within the REMD 
algorithm output sample set. 

The experiments on object observation manifolds are con- 
ducted similarly. We construct and sample the observation 
manifold of each object individually. Experiments are repeated 
5 times for each class with different random initializations. The 
results are again averaged over all realizations and all objects. 



Fig. 9(a) shows the registration errors obtained with the 



samplings and Fig. 9(b) shows the percentage of test images 
with the best projection approximations within the REMD 
output, random and the regular sample sets. The results are in 
accordance with the results obtained on pattern transformation 
manifolds. Note that although the intrinsic dimensions of man- 
ifolds are the same in the two experimental setups, the typical 
number of samples required for accurately representing the 
pattern transformation manifolds and the object observation 
manifolds is quite different in these experiments. This is 
due to the differences in the type and range of geometric 
transformations that generate the manifolds. The fact that 
the object observation manifolds are defined over a relatively 
small parameter domain makes it possible to represent them 
with fewer samples compared to the pattern transformation 
manifolds for similar performance. 

C. Results on transformation-invariant classification 

In this part, we evaluate the performances of the discretiza- 



tion approaches in Section IV in transformation-invariant 



classification. In all of the pattern transformation manifold 
experiments, the image set of each object in the database is 
regarded as a different signal class. Similarly, in the synthetical 
objects database, the rendered images of each class of objects 
are considered to belong to a separate signal class. Only 
training images are available to the sampling algorithms, and 
the classification performance is measured on test images. 
Once the sample sets are obtained, the classification of a test 
image is performed by assigning the label of the manifold 
sample with smallest distance. In all of the following figures, 
the correct classification rates of test images are plotted 
in percentage with respect to the number of samples per 
manifold. All experiments are repeated 10 times with different 
random algorithm initializations and averaged. 

First, we compare the REMD algorithm, the random dis- 
cretization and the regular discretization in the parameter 



transformation manifold of each class is sampled individually 
via the REMD algorithm, randomly and regularly, where an 
equal number of samples are selected on each manifold. The 
results obtained on pattern transformation manifolds and ob- 



ject observation manifolds are plotted respectively in Fig. lQ(a) 



and Fig. |lQ(b)| The plots indicate that in both setups, the 
REMD output sample set has higher classification performance 
compared to the random and regular discretizations. This is in 
agreement with the results of the registration experiments of 



Section V-B confirming the dependency of the transformation- 
invariant classification performance on the accuracy of man- 



ifold distance estimation. When the two plots in Fig. 10(a) 



and Fig. lQ(b) are compared, it is seen that the classification 
rate improvement introduced by REMD or by increasing 
the number of samples is higher in pattern transformation 
manifolds. This can be explained by the difference between 
the two setups. In object observation manifold experiments 
there are several object models belonging to the same class. 
Therefore, space points have a relatively large deviation from 
the manifold of the representative object. This deviation is 
smaller in pattern transformation manifolds as the space points 
of a specific class are the images of the same object. 

Then we search the efficiency of jointly optimizing all 
manifold samples in comparison with sampling each mani- 
fold individually. For this purpose, we first select an equal 
number of samples from each manifold independently with 
the REMD algorithm as in the previous experiment. Then 
we apply an additional stage of joint optimization, where 
we optimize the samples from all classes together via two 
alternative approaches. In the first approach, we optimize 
the output sample set of the REMD algorithm further with 
the CMD algorithm (marked simply as CMD in the plots). 
In the second one we again begin with the REMD output 
sample set, but then perform the joint optimization of manifold 
samples using a simulated annealing algorithm where we 
define the cost function as the classification error e in ([16]). We 
name this approach Manifold Discretization with Simulated 
Annealing (MDSA). The simulated annealing algorithm is 
based on seeking the global optimum by trying random search 
directions, whereas the CMD algorithm has a more restricted 
search domain. Therefore, the results with MDSA are pro- 
vided as a benchmark for the evaluation of the efficiency 
of CMD. The classification rates are plotted in Fig. ll(a)| 



for pattern transformation manifolds and in Fig. 11(b) for 
object observation manifolds. The results show that the joint 
optimization of manifold samples after the individual sampling 
stage brings a significant improvement on the classification 
rate. This is consistent with the expectation that the relative 
characteristics of manifolds, i.e., their structures with respect 
to each other, should also be taken into account as well as 
their individual characteristics in classification. Moreover, the 
performances of the sample sets obtained by CMD and MDSA 
are close to each other. This shows that CMD is an effective 
constructive algorithm. The slight superiority of MDSA to 
CMD is justifiable in the sense that the CMD algorithm 
performs a one-dimensional search in the parameter domain 
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at each update step, whereas in simulated anneahng the search 
space is full dimensional in the parameter domain. 

Finally, in a third experiment we examine the effect of the 
uneven distribution of the total sample budget to different 
manifolds. We compare the performances of the CMD algo- 
rithm with equal budget distribution to different manifolds, 
DMD and MDPA, which have been discussed in Section [TV-Cl 
In order to test CMD, we first select an equal number of 
samples from each manifold independently with the REMD 
algorithm, and then optimize the output of REMD in a further 
stage with CMD as in the previous experiment. We apply 
the same procedure for DMD as well; it is initialized with 
the output of REMD with equally distributed samples, where 
the movement of samples between different manifolds is 
allowed afterwards to optimize the distribution of the sample 
budget. The correct classification rates obtained with the three 
sampling approaches are plotted with respect to the average 



number of samples selected per manifold in Fig. 12(a) and 
Fig. |12(b)| respectively for pattern transformation and object 
observation manifolds. The results suggest that an uneven dis- 
tribution of the sample budget to different manifolds according 
to their geometric properties may improve the classification 
accuracy when compared to the equal distribution of samples. 
It is seen that the performances of DMD and MDPA are close 
to each other in the object observation manifolds experiment. 
However, in the pattern transformation manifolds experiment, 
the number of available training images per manifold sample is 
much smaller, which has a negative influence on the efficiency 
of budget distribution in the first stage of progressive sample 
deletion in MDPA. 
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D. Discussion of Results 

Here we make an interpretation of our experimental results 
from the perspective of the trade-off between computational 
complexity and performance. The main motivation behind this 
work is the difficulty of the exact computation of the manifold 
distance. The state-of-the-art methods accomplishing mani- 
fold distance computation are considerably demanding. For 
instance, the algorithm proposed in |4| involves a complexity 
of 0(iir • ni • 712), where K is the number of atoms used in the 
decomposition of the reference pattern and ni xn2 is the image 
resolution, while a similar algorithm complexity is reported 
in 0. On the other hand, in order to estimate the manifold 
distance we propose the utilization of a suitable manifold grid 
that is to be determined offline. Once the grid is obtained, 
the manifold distance estimation is simplified merely to the 
computation of the norms of the difference vectors between the 
query signal and the samples, which clearly reduces the cost of 
distance estimation significantly. Meanwhile, it is not easy to 
draw a general conclusion in the comparison of the accuracies 
of manifold distance computation algorithms and our grid 
approach. This is highly dependent on the algorithms under 
comparison. For instance, the algorithm in |4| is guaranteed 
to find the global solution in the projection of query images 
onto pattern transformation manifolds, resulting in a perfectly 
accurate distance computation. However, methods such as O, 
(3, El do not have such optimal performance guarantees. In 



Fig. 10. Classification results obtained by sampling class representative 
manifolds individually 



the approximation of the manifold distance with a grid, the 
registration accuracy is clearly dependent on the number of 
samples, and the registration error asymptotically approaches 
zero as the number of samples increases. We also note that, in 
presence of large geometric transformations the grid approach 
may have an advantage over algorithms based on tangent 
distance, which are susceptible to local minima. Considering 
these, we conclude that sample-based approaches achieve a 
compromise between accuracy and computational effort. 

Now, let us turn to the complexity of the discretization 
process. To start with, the REMD algorithm is in principle 
an adaptation of the LBG vector quantization algorithm if TTIl 
to the manifold-modeled signal case. Therefore the main 
complexity of the algorithm is 0{N), where N is the number 
of selected samples. As CMD is an extension of REMD to 
multiple manifolds, it has the same order of complexity with 
respect to the total number of samples. In the case that the 
centroids are estimated via training images, the complexity of 
the discretization algorithm also has a first-order dependence 
on the number of training images 1 18 1. Also, a practical matter 
of concern is the estimation of the projections of the centroids 
onto manifolds, which we overcome by the aid of a dense 
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Fig. 11. Effect of the joint optimization of samples on classification accuracy 



Fig. 12. Effect of the uneven distribution of sample budget on classification 
accuracy 



grid. However, the manner of the determination of projections 
is out of the scope of this work, as a variety of solutions may 
be appHcable depending on the type of the treated signals. 

Finally, we note that the classification performance of CMD, 
which has a refined search space, is fairly close to that of 
MDSA where the correct classification rate is optimized by 
simulated annealing. Comparing these two approaches with 
respect to their convergence rates, we have seen that CMD 
terminates in a much less number of iterations, which is a 
result of its capability of assessing the proper search directions. 
Yet, the overall running time of the CMD algorithm depends 
on the computational time required by the projection of space 
points onto manifolds. The speed of the required registration 
block depends on the type of transformations involved. Ef- 
ficient solutions exist for certain geometric transformations. 
For instance, the phase correlation method |19 | is a well- 
known and fast technique that recovers image translations. 
We remark also that image registration is an active research 
field and recent works such as f20l are promising for the 
generalization of such techniques to handle a wider range of 
geometric transformations. 



VI. Conclusion 

We have studied the sampling of signal manifolds with 
known parameterization. We present a discretization solution 
for a single manifold based on registration accuracy. Then 
we generalize the problem to the discretization of multiple 
signal manifolds that represent different classes of signals, and 
propose a method for the joint optimization of all manifold 
samples for the improvement of classification performance. We 
also discuss possible ways of optimizing the distribution of a 
fixed sample budget to different class representative manifolds 
in order to improve the classification accuracy. We test the 
proposed sampling approaches on pattern transformation man- 
ifolds and object observation manifolds. Experimental results 
indicate that the registration accuracy of the distance-based 
sampling is considerably higher than random and regular sam- 
plings. The consideration of the relative structures of different 
manifolds in the discretization improves the classification 
performance significantly when compared to the independent 
discretization of each manifold. Moreover, distributing the 
total sample budget unequally to the manifolds in a manner 
that takes account of their different characteristics may also 
bring an improvement. The results reveal the potential of our 



13 



work to find various appHcation areas in the treatment of 
parametrizable signal sets. 
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